Image processing apparatus, moving apparatus, method, and program

ABSTRACT

Configuration, in which images output to a display unit are switched and displayed in accordance with the behavior of a driver, such as movements of the head of the driver, is achieved. Driver information indicating the behavior of the driver of a moving apparatus and images captured by a plurality of cameras that images a situation around the moving apparatus from different viewpoints are input. The images output to the display unit are switched in accordance with the driver information. The plurality of cameras is, for example, a plurality of rear cameras installed in the rear of the moving apparatus. For example, a direction of the face or line-of-sight of the driver is detected. An image in a direction corresponding to the detected direction of the face or line-of-sight of the driver is selected as an output image, and displayed on the display unit. Alternatively, an image in a direction indicated by a gesture of the driver is selected, and displayed on the display unit.

TECHNICAL FIELD

The present disclosure relates to an image processing apparatus, amoving apparatus, a method, and a program. More specifically, thepresent disclosure relates to an image processing apparatus, a movingapparatus, a method, and a program that controls a display image to beoutput to a display unit (monitor) at a driver seat on the basis of animage captured by a camera mounted in a moving apparatus such as anautomobile.

BACKGROUND ART

These days, camera monitor systems (CMSs) can be mounted, and the CMSsare started to be actually introduced to automobiles. In the CMSs, animage captured by a camera mounted in an automobile is displayed on amonitor at a driver seat, and the situation around the automobile, forexample, the situation of the rear (rear direction) of the automobilecan be checked.

For example, Patent Document 1 (WO 2018/012299) discloses a cameramonitor system (CMS) in a host vehicle. In the CMS, an image of the rearof a vehicle captured by a rear camera installed in the rear of thevehicle, images of the right rear and left rear captured by side camerasinstalled at rearview mirror positions on the right and left sides ofthe vehicle, or a composite image thereof are displayed on a displayunit.

CITATION LIST Patent Document Patent Document 1: WO 2018/012299 SUMMARYOF THE INVENTION Problems to be Solved by the Invention

A camera monitor system (CMS) enables a surrounding state, which hasbeen checked with a back view mirror such as a so-called traditionalrearview mirror and side mirror of a car, to be checked by using acamera captured image.

For example, three images of a rear captured image, a left rear capturedimage, and a right rear captured image are individually displayed on adisplay unit (monitor) capable of being observed by a driver. The rearcaptured image is captured by a rear camera that images the rear of avehicle. The left rear captured image and a right rear captured imageare respectively captured by a left side camera and a right side camerainstalled in the vicinity of left and right side mirror positions of thevehicle. Alternatively, a single composite image generated byappropriately synthesizing these three images is displayed on themonitor.

In a case where one continuous composite image is generated anddisplayed from three images captured by the rear camera and left andright side cameras in such an image display system, however, a subjectcaptured in any of the three images sometimes disappear, and is notdisplayed. This is because the images captured by the three cameras ofthe rear camera, the L side camera, and the R side camera have differentviewpoints, and this is caused by, for example, processing of conversionto an image from one viewpoint, for example, a viewpoint of the rearcamera, at the time when a composite image is generated.

If a composite image is generated with respect to the viewpoint of therear camera at the time when a composite image is generated, a subjectcaptured by a left side camera and a right side camera, such as, forexample, a motorcycle behind a large vehicle in the immediate rear and avehicle approaching at a high speed from a distance, disappears from thecomposite image. This is because, for example, the motorcycle is in aregion behind the large vehicle, that is, an occlusion region from theviewpoint of the rear camera.

The present disclosure has been made in view of, for example, theabove-described problem, and an object of thereof is to provide an imageprocessing apparatus, a moving apparatus, a method, and a program inwhich a driver can almost continuously (seamlessly) check images of therear side observed from various viewpoints similarly to the case ofchecking a traditional rearview mirror and side mirror by sequentiallyswitching and displaying images captured by a plurality of camerashaving different imaging viewpoints in accordance with the behavior ofthe driver, for example, movements of the head.

Solutions to Problems

A first aspect of the present disclosure is in an image processingapparatus including a data processing unit that executes image displaycontrol of:

inputting

driver information indicating behavior of a driver of a moving apparatusand

images captured by a plurality of cameras that images a situation aroundthe moving apparatus from different viewpoints; and

switching an image output to a display unit to any of the imagescaptured by the plurality of cameras or a virtual viewpoint compositeimage in accordance with the driver information.

Moreover, a second aspect of the present disclosure is in a movingapparatus including:

a plurality of cameras that images a situation around the movingapparatus from different viewpoints;

a driver information detection unit that detects driver informationindicating behavior of a driver of the moving apparatus; and

a data processing unit that inputs the driver information and imagescaptured by the plurality of cameras, and switches an image output to adisplay unit to any of the images captured by the plurality of camerasor a virtual viewpoint composite image in accordance with the driverinformation.

Moreover, a third aspect of the present disclosure is in an imageprocessing method executed in an image processing apparatus,

in which a data processing unit executes image display control of:

inputting

driver information indicating behavior of a driver of a moving apparatusand

images captured by a plurality of cameras that images a situation aroundthe moving apparatus from different viewpoints; and

switching an image output to a display unit to any of the imagescaptured by the plurality of cameras or a virtual viewpoint compositeimage in accordance with the driver information.

Moreover, a fourth aspect of the present disclosure is in a displayimage control method executed in a moving apparatus, including:

an image capturing step in which a plurality of cameras images asituation around the moving apparatus from different viewpoints;

a driver information detection step in which a driver informationdetection unit detects driver information indicating behavior of adriver of the moving apparatus; and

an image display control step in which a data processing unit inputs thedriver information and images captured by the plurality of cameras, andswitches an image output to a display unit to any of the images capturedby the plurality of cameras or a virtual viewpoint composite image inaccordance with the driver information.

Moreover, a fifth aspect of the present disclosure is in a program forexecuting image processing in an image processing apparatus, causing adata processing unit to execute image display control of:

inputting

driver information indicating behavior of a driver of a moving apparatusand

images captured by a plurality of cameras that images a situation aroundthe moving apparatus from different viewpoints; and

switching an image output to a display unit to any of the imagescaptured by the plurality of cameras or a virtual viewpoint compositeimage in accordance with the driver information.

Note that the program of the present disclosure can be provided by astorage medium or a communication medium capable of providing variousprogram codes in a computer readable format to an information processingapparatus and a computer system capable of executing various programcodes. Processing in accordance with a program is performed on aninformation processing apparatus and a computer system by providing sucha program in a computer-readable format.

Other objects, features, and advantages of the present disclosure willbe apparent from more detailed description based on the later-describedembodiments of the present disclosure and the accompanying drawings.Note that a system in the specification has configuration of a logicalset of a plurality of apparatuses. The system is not limited to a systemin which apparatuses having each configuration are placed in the samehousing.

Effects of the Invention

According to the configuration of one embodiment of the presentdisclosure, configuration, in which images output to a display unit areswitched and displayed in accordance with the behavior of a driver, suchas movements of the head of the driver, is achieved.

Specifically, for example, driver information indicating the behavior ofa driver of a moving apparatus and images captured by a plurality ofcameras that images the situation around the moving apparatus fromdifferent viewpoints are input. Images output to the display unit areswitched in accordance with driver information. The plurality of camerasis, for example, a plurality of rear cameras installed in the rear ofthe moving apparatus. For example, a direction of the face orline-of-sight of the driver is detected. An image in a directioncorresponding to the detected direction of the face or line-of-sight ofthe driver is selected as an output image, and displayed on the displayunit. Alternatively, an image in a direction indicated by a gesture ofthe driver is selected, and displayed on the display unit.

The configuration achieves configuration in which images output to thedisplay unit are switched and displayed in accordance with the behaviorof the driver, such as movements of the head of the driver.

Note that the effects described in the specification are merelyillustration and not limitation, and additional effects may beexhibited.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 outlines a camera monitor system (CMS) mounted in an automobile.

FIG. 2 illustrates an example of usage of the CMS.

FIG. 3 illustrates an example of usage of the CMS.

FIG. 4 illustrates an example of usage of the CMS.

FIG. 5 illustrates the configuration of a moving apparatus and an imageprocessing apparatus of the present disclosure and processing therein.

FIG. 6 illustrates the configuration of the moving apparatus and theimage processing apparatus of the present disclosure and the processingtherein.

FIG. 7 illustrates the configuration of the moving apparatus and theimage processing apparatus of the present disclosure and the processingtherein.

FIG. 8 illustrates an example of image display on a display unit.

FIG. 9 illustrates an example of the image display on the display unit.

FIG. 10 illustrates an example of the image display on the display unit.

FIG. 11 illustrates an example of the image display on the display unit.

FIG. 12 illustrates the configuration of the moving apparatus and theimage processing apparatus and the processing therein.

FIG. 13 illustrates an example of the image display on the display unit.

FIG. 14 illustrates an example of the image display on the display unit.

FIG. 15 illustrates an example of camera arrangement in the movingapparatus.

FIG. 16 illustrates an example of camera arrangement in the movingapparatus.

FIG. 17 illustrates an example of camera arrangement and display imagesin the moving apparatus.

FIG. 18 illustrates an example of camera arrangement and display imagesin the moving apparatus.

FIG. 19 illustrates an example of camera arrangement and display imagesin the moving apparatus.

FIG. 20 illustrates an example of camera arrangement and display imagesin the moving apparatus.

FIG. 21 illustrates an example of camera arrangement and display imagesin the moving apparatus.

FIG. 22 illustrates an example of camera arrangement in the movingapparatus.

FIG. 23 is a flowchart illustrating a processing sequence executed bythe image processing apparatus.

FIG. 24 is a flowchart illustrating the processing sequence executed bythe image processing apparatus.

FIG. 25 illustrates an example of the hardware configuration of theimage processing apparatus.

MODE FOR CARRYING OUT THE INVENTION

Details of an image processing apparatus, a moving apparatus, a method,and a program of the present disclosure will be described below withreference to the drawings. Note that the description will be given inaccordance with the following items.

1. Outline of In-vehicle Camera Monitor System (CMS)

2. Configuration Example of Image Processing Apparatus and MovingApparatus of Present Disclosure

3. Other Embodiments

4. Image Processing Apparatus and Sequence of Processing Executed byMoving Apparatus

5. Configuration Example of Image Processing Apparatus

6. Summary of Configuration of Present Disclosure

1. Outline of In-vehicle Camera Monitor System (CMS)

First, a camera monitor system (CMS) mounted in an automobile will beoutlined with reference to FIG. 1 and subsequent figures.

FIG. 1 outlines a CMS mounted in an automobile 10. The CMS displays animage captured by a camera installed on an automobile on a display unitthat a driver can observe.

The automobile 10 in FIG. 1 includes a rear camera 11, a left sidecamera 12, and a right side camera 13. The rear camera 11 is installedin the rear of the automobile 10. The left side camera 12 is installedon the left side of the automobile 10. The right side camera 13 isinstalled on the right side of the automobile 10. Each of these camerascaptures an image behind the automobile 10, observed from theinstallation position of each camera.

Images captured by these cameras or a composite image generated on thebasis of these captured images are displayed on a display unit 20 insidethe automobile 10. A driver of the automobile 10 can check the statebehind the automobile 10 by looking at a display image on the displayunit 20.

A specific image example displayed on the display unit 20 will bedescribed with reference to FIG. 2 and subsequent figures. FIG. 2illustrates the state behind the automobile 10. In FIG. 2, theautomobile 10 is traveling in a direction of an arrow (forward) in thefigure.

In this state, the three cameras of the rear camera 11, the left sidecamera 12, and the right side camera 13 capture an image with eachcamera position as a viewpoint position.

FIG. 3 illustrates examples of images, which the driver of theautomobile 10 can check, displayed on the display unit 20. FIG. 3illustrates an example in which images captured by three cameras of therear camera 11, the left side camera 12, and the right side camera 13are individually displayed on the display unit 20. On the display unit20, a rear camera image 21 captured by the rear camera 11 is displayedin the center of the display unit 20, a left side camera image 22captured by the left side camera 12 is displayed on the left side of thedisplay unit 20, and a right side camera image 23 captured by the rightside camera 13 is displayed on the right side of the display unit 20.

The driver can check the surrounding situation in a manner similar tothe case of looking at a rearview mirror and right and left side mirrorsby looking at these three images.

In a case where three images are displayed side by side in this way,however, the driver mainly pays attention only to the central imagecorresponding to the rearview mirror. If the driver is not careful ofthe right and left images, the driver tends to overlook informationmainly reflected in the right and left images.

In the example in FIG. 3, a bicycle reflected in the left side cameraimage 22 is hidden behind a large vehicle in the central rear cameraimage 21, and almost invisible. In such a case, if the driver paysattention only to the rear camera image 21 displayed in the center ofthe display unit 20, the driver is at risk of being unaware of thebicycle.

Although FIG. 3 illustrates an example in which images captured by thecameras are individually displayed on the display unit 20, aconfiguration, in which a composite image is generated by combiningthese three images to one image and displayed on the display unit 20,has been proposed and used. FIG. 4 illustrates an example of display ofa composite image.

FIG. 4 illustrates one example of a composite image 25 displayed on thedisplay unit 20. The composite image 25 is generated with the viewpointof the rear camera 11 as a reference. The rear camera 11, the left sidecamera 12, and the right side camera 13 capture images from differentviewpoints, so that, for example, processing of converting a viewpointposition of each image, processing of converting the size of an object(subject), and processing of adjusting a position are necessary forgenerating one composite image. Specifically, image conversionprocessing such as, for example, affine transformation is performed.

Multiple images are joined by such image conversion processing togenerate one composite image. FIG. 4 illustrates an example of acomposite image generated by converting images captured by the left sidecamera 12 and the right side camera 13 into a captured image from theviewpoint of the rear camera 11. In a case where such a composite imageis generated, however, a subject captured by the left side camera or theright side camera may disappear.

Although the bicycle serving as a subject can be clearly confirmed inthe above-described left side camera image 22 in FIG. 3, the bicycle ishidden behind the vehicle, and almost invisible in the composite image25 in FIG. 4. This is because the bicycle is in a region behind thelarge vehicle in front of the vehicle, that is, an occlusion region fromthe viewpoint of the rear camera.

The driver cannot confirm the bicycle approaching from behind even iflooking at the composite image 25. If the driver drives in a manner of,for example, suddenly turning left, an accident such as contact againstthe bicycle approaching from behind may occur.

2. Configuration Example of Image Processing Apparatus and MovingApparatus of Present Disclosure

Next, a configuration example of the image processing apparatus and themoving apparatus of the present disclosure will be described withreference to FIG. 4 and subsequent figures. The present disclosuresolves the problem described with reference to FIG. 4, for example.Specifically, for example, images captured by a plurality of camerashaving different imaging viewpoints are sequentially switched anddisplayed in accordance with the behavior of a driver, such as, forexample, movements of a head, face, and line-of-sight.

FIG. 5 illustrates one configuration example of the automobile 10(moving apparatus) of the present disclosure. The automobile 10 in FIG.5 includes the left side camera 12 on the left side of the automobile 10and the right side camera 13 on the right side of the automobile 10, ina manner similar to that described above with reference to FIG. 1. Threerear cameras, that is, a rear central (C) camera 31, a rear left (L)camera 32, and a rear right (R) camera 33 are installed in the rear ofthe automobile 10. The rear C camera 31 is installed in the center ofthe rear of the automobile 10. The rear L camera 32 is installed on theleft side of the rear of the automobile 10. The rear R camera 33 isinstalled on the right side of the rear of the automobile 10.

Note that, although three cameras are installed in the rear of theautomobile 10 in the example, a plurality of various cameras other thanthree can be used. An example of using three or more multiple cameraswill be described later.

FIG. 6 illustrates an example of imaging regions of three camerasinstalled in the rear of the automobile 10. FIG. 6 illustrates theimaging regions of three cameras of the rear central (C) camera 31, therear left (L) camera 32, and the rear right (R) camera 33. The rear Ccamera 31 is installed in the center of the rear of the automobile 10.The rear L camera 32 is installed on the left side of the rear of theautomobile 10. The rear R camera 33 is installed on the right side ofthe rear of the automobile 10.

As illustrated in FIG. 6, the three cameras have setting in which therear of the automobile is imaged from different viewpoints, and parts ofthe imaging region overlap with each other. In the configuration of thepresent disclosure, images captured by a plurality of cameras havingsuch different imaging viewpoints are sequentially switched anddisplayed in accordance with the behavior of the driver, for example,movements of the head. The movements of the head are detected fordetecting a movement comparable to action and motion of the driver atthe time of looking at a traditional rearview mirror and side mirror andusing the sign as a trigger for switching display.

FIG. 7 illustrates an imaging environment similar to that in FIG. 2described above. The automobile 10 is traveling in the direction of anarrow (forward) in the figure.

A large vehicle is traveling behind the automobile 10, and a bicycle istraveling in the left rear of the large vehicle.

FIG. 8 illustrates examples of images, which the driver of theautomobile 10 can check, displayed on the display unit 20. The left sidecamera image 22 captured by the left side camera 12 is displayed on theleft side of the display unit 20. The right side camera image 23captured by the right side camera 13 is displayed on the right side ofthe display unit 20.

An image display region in the central part of the display unit 20 isset as a rear-camera-image switching display region 41. Images capturedby three cameras, that is, the rear central (C) camera 31, the rear left(L) camera 32, and the rear right (R) camera 33, installed in the rearof the automobile 10 are sequentially switched and displayed on therear-camera-image switching display region 41 in accordance with thebehavior of the driver, for example, movements of the head. The rear Ccamera 31 is installed in the center of the rear of the automobile 10.The rear L camera 32 is installed on the left side of the rear of theautomobile 10. The rear R camera 33 is installed on the right side ofthe rear of the automobile 10.

An example of image display on the rear-camera-image switching displayregion 41, that is, an image switching display example will be describedwith reference to FIG. 9.

FIG. 9 illustrates correspondence data between movements of the head ofa driver 50 (direction of the face or line-of-sight) and the example ofimage display on the rear-camera-image switching display region 41.Three examples of image display are given below.

(R) Display Example of Rear R Camera Captured Image

(C) Display Example of Rear C Camera Captured Image

(L) Display Example of Rear L Camera Captured Image

FIG. 9 illustrates (R) display example of a rear R camera captured imagein a case where the face or line-of-sight of the driver 50 is in theright direction. In this way, in a case where the driver 50 of theautomobile 10 looks at the right, an image captured by the rear right(R) camera 33, which is installed on the right side of the rear of theautomobile 10, is displayed on the rear-camera-image switching displayregion 41 of the display unit 20.

Furthermore, FIG. 9 illustrates (C) display example of a rear C cameracaptured image in a case where the face or line-of-sight of the driver50 is substantially in the front direction. In this way, in a case wherethe driver 50 of the automobile 10 looks at the front, an image capturedby the rear central (C) camera 31, which is installed in the center ofthe rear of the automobile 10, is displayed on the rear-camera-imageswitching display region 41 of the display unit 20.

Moreover, FIG. 9 illustrates (L) display example of a rear L cameracaptured image in a case where the face or line-of-sight of the driver50 is in the left direction. In this way, in a case where the driver 50of the automobile 10 looks at the left, an image captured by the rearleft (L) camera 32, which is installed on the left side of the rear ofthe automobile 10, is displayed on the rear-camera-image switchingdisplay region 41 of the display unit 20.

Note that, in a case where the driver 50 exhibits behavior of, forexample, first looking at the right, then looking at the front, andfinally looking at the left, display images on the rear-camera-imageswitching display region 41 of the display unit 20 are sequentiallyswitched as follows. First, an image captured by the rear right (R)camera 33 is displayed, switched to an image captured by the rearcentral (C) camera 31, and then switched to display an image captured bythe rear left (L) camera 32.

A driver information detection unit that detects the behavior of thedriver 50 is installed in the automobile 10. An image processingapparatus of the automobile 10 performs control to switch display imageson the display unit 20 on the basis of detection information from thedriver information detection unit.

An image of a bicycle can be clearly confirmed in (L) display example ofthe rear L camera captured image described on the left side among threeimages in FIG. 9. This is because the rear left (L) camera 33 capturesan image from the left side of the rear of the automobile 10, and cancapture an image of the bicycle that is not hidden behind the largevehicle in front of the bicycle.

The driver 50 can clearly confirm that the bicycle is behind the largevehicle by looking at the image.

Note that, although each image in FIG. 9 may be displayed on a centralregion between the left side camera image 22 and the right side cameraimage 23 together with these two images as in the display example of thedisplay unit 20 described above with reference to FIG. 8, only an imagein FIG. 9 may be independently displayed on the display unit 20 inaccordance with the direction of the face or line-of-sight of the driver50 without displaying the left side camera image 22 and the right sidecamera image 23.

Note that FIG. 9 illustrates setting of switching display images on thedisplay unit 20, that is, images on the rear-camera-image switchingdisplay region 41, which is a central region of the display unit 20 inFIG. 8, in accordance with the direction of the face or line-of-sight ofthe driver 50, the position of the rear-camera-image switching displayregion 41 may be changed in accordance with the direction of the face orline-of-sight of the driver 50, for example. That is, the position ofthe rear-camera-image switching display region 41 is changed to aposition that matches the direction of the face or line-of-sight of thedriver 50.

The image position change display example will be described withreference to FIG. 10. Similarly to FIG. 9, FIG. 10 illustratescorrespondence data between movements of the head of a driver 50(direction of the face or line-of-sight) and the example of imagedisplay on the rear-camera-image switching display region 41. Threeexamples of image display are given below.

(R) Display Example of Rear R Camera Captured Image

(C) Display Example of Rear C Camera Captured Image

(L) Display Example of Rear L Camera Captured Image

The display unit 20 in FIG. 10 is set on an upper part of a front panelof an automobile, and has a display region that is long in the right andleft direction.

FIG. 10 illustrates (L) display example of the rear L camera capturedimage in a case where the face or line-of-sight of the driver 50 is inthe left direction. In a case where the driver 50 looks at the left, animage captured by the rear left (L) camera 32, which is installed on theleft side of the rear of the automobile 10, is displayed on a left endpart of the display unit.

Furthermore, FIG. 10 illustrates (C) display example of the rear Ccamera captured image in a case where the face or line-of-sight of thedriver 50 is substantially in the front direction. In a case where thedriver 50 looks at the front, an image captured by the rear central (C)camera 31, which is installed in the center of the rear of theautomobile 10, is displayed in the central part of the display unit.

Furthermore, FIG. 10 illustrates (R) display example of the rear Rcamera captured image in a case where the face or line-of-sight of thedriver 50 is in the right direction. In a case where the driver 50 looksat the right, an image captured by the rear right (R) camera 33, whichis installed on the right side of the rear of the automobile 10, isdisplayed on a right end part of the display unit.

FIG. 10 illustrates an example in which the position of therear-camera-image switching display region 41 is changed to a positionthat matches the direction of the face or line-of-sight of the driver50. Such display position control allows the driver 50 to display animage which the driver 50 wants to see in the direction of line-of-sightof his/her own. Consequently, the driver 50 can certainly confirm theimage in the direction in which the driver 50 is paying attention, whilekeeping the line-of-sight in the direction.

Note that the control of the image display position is also executed onthe basis of the detection information from the driver informationdetection unit, which detects the behavior of the driver 50, installedin the automobile 10.

Moreover, separately from the direction of the face or line-of-sight ofthe driver 50, display images may be changed in accordance withindication with, for example, a head movement sequence, a hand, or afinger of the driver 50.

An example of the configuration in which display images are changed inaccordance with indication with the hand or finger of the driver 50 willbe described with reference to FIG. 11.

FIG. 11 illustrates (L) display example of the rear L camera capturedimage in a case where a finger of driver 50 indicates the leftdirection. In a case where the finger of the driver 50 indicates theleft, an image captured by the rear left (L) camera 32, which isinstalled on the left side of the rear of the automobile 10, isdisplayed on the display unit 20.

Furthermore, FIG. 11 illustrates (C) display example of the rear Ccamera captured image in a case where the finger of the driver indicatessubstantially the front. In a case where the finger of the driver 50indicates the front, an image captured by the rear central (C) camera31, which is installed in the center of the rear of the automobile 10,is displayed on the display unit 20.

Moreover, FIG. 11 illustrates (R) display example of the rear R cameracaptured image in a case where the finger of the driver 50 indicates theright. In a case where the finger of the driver 50 indicates the right,an image captured by the rear right (R) camera 33, which is installed onthe right side of the rear of the automobile 10, is displayed on thedisplay unit 20.

FIG. 11 illustrates an example in which images displayed on therear-camera-image switching display region 41 are changed in accordancewith indication (gesture) with a hand or finger of the driver 50. Notethat the motion sequence of a head may be detected as a gesture. Suchcontrol enables the driver 50 to change images without significantlychanging the line-of-sight direction.

FIG. 12 illustrates a configuration example for image switching displaycontrol provided in the automobile 10, which is a moving apparatus. Asillustrated in FIG. 12, the automobile 10 includes a driver informationdetection unit 52 that detects the direction of the face orline-of-sight of the driver 50 or indication (gesture) with a hand orfinger. Specifically, the driver information detection unit 52 includes,for example, a camera or a movement sensor. The detection informationfrom the driver information detection unit 52 is input to a dataprocessing unit 60.

The data processing unit 60 performs processing of switching imagesdisplayed on the display unit 20, and changes a display position on thebasis of the detection information from the driver information detectionunit 52.

Note that a user (driver 50) may perform operation of switching imagesdisplayed on the display unit 20. For example, as illustrated in FIG.13, for example, a user operation unit may be displayed on an operationunit set in the vicinity of the display unit 20 or a display unit oftouch panel type. The user may operate the operation unit to switchdisplay images.

(L) display example of the rear L camera captured image in FIG. 13 is anexample of a display image on the display unit 20 in a case where theuser moves the operation unit to the left. In this case, an imagecaptured by the rear left (L) camera 32, which is installed on the leftside of the rear of the automobile 10, is displayed on the display unit20.

(C) display example of the rear C camera captured image is an example ofa display image on the display unit 20 in a case where the user movesthe operation unit to the center. In this case, an image captured by therear central (C) camera 31, which is installed in the center of the rearof the automobile 10, is displayed on the display unit 20.

(R) display example of the rear R camera captured image is an example ofa display image on the display unit 20 in a case where the user movesthe operation unit to the right. In this case, an image captured by therear right (R) camera 33, which is installed on the right side of therear of the automobile 10, is displayed on the display unit 20.

Note that setting similar to that described above with reference to FIG.9 can be adopted. In the setting, each image in FIGS. 9 to 13 isdisplayed together with the left side camera image 22 and the right sidecamera image 23 as the display example of the display unit 20 describedabove with reference to FIG. 8. Furthermore, only an image in FIGS. 9 to13 may be independently displayed on the display unit 20 in accordancewith the direction of the face or line-of-sight and indication of thedriver 50 without displaying the left side camera image 22 and the rightside camera image 23.

The image switching display example described with reference to FIGS. 9to 13 is a configuration example in which images captured by threecameras of, that is, the rear left (L) camera 32, the rear central (C)camera 31, and the rear right (R) camera 33 installed in the rear of theautomobile 10 are switched and displayed on the basis of the position ofthe head/face, line-of-sight direction, or indication of the driver.

Moreover, an image captured by the left side camera 12 and an imagecaptured by the right side camera 13 may be added to images captured bythese three cameras as a target of switching images.

The specific example will be described with reference to FIG. 14.

FIG. 14 illustrates an example of a display image switching sequence ofthe display unit 20 based on, for example, the direction of the face orline-of-sight of the driver 50, indication, or input to an operationunit.

For example, as the direction of the face or line-of-sight of the driver50 changes from right to left, display images on the display unit 20 aresequentially switched in the following order.

(1) Right Side Camera Image Captured by Right Side Camera 13

(2) Rear R Camera Captured Image Captured by Rear R Camera 33

(3) Rear C Camera Captured Image Captured by Rear C Camera 31

(4) Rear L Camera Captured Image Captured by Rear L Camera 32

(5) Left Side Camera Captured Image Captured by Left Side Camera 12

Moreover, as the direction of the face or line-of-sight of the driver 50changes from left to right, display images on the display unit 20 aresequentially switched in the order of (5) to (1) above.

3. Other Embodiments

The embodiment described with reference to FIGS. 5 to 14 is aconfiguration example in which images captured by three cameras of, thatis, the rear left (L) camera 32, the rear central (C) camera 31, and therear right (R) camera 33 installed in the rear of the automobile 10 areswitched and displayed on the basis of the direction of the face orline-of-sight of a driver, gesture indication with, for example, a head,or input to the operation unit.

Any multiple number other than three of cameras can be installed in therear of the automobile 10.

FIG. 15 illustrates an example in which five cameras are installed inthe rear of the automobile 10. FIG. 15 illustrates an example similar tothe embodiment described with reference to FIGS. 5 to 14. In theexample, the rear left (L) camera 32, the rear central (C) camera 31,and the rear right (R) camera 33 are installed. Moreover, a rear centralleft (CL) camera 34 and a rear central right (CR) camera 35 areprovided. The rear CL camera 34 captures an image from a middleviewpoint between the rear L camera 32 and the rear C camera 31. Therear CR camera 35 captures an image from a middle viewpoint between therear C camera 31 and the rear R camera 33.

FIG. 16 illustrates an example of imaging regions of these five cameras.FIG. 16 illustrates the imaging regions of the five cameras of the rearleft (L) camera 32, the rear central left (CL) camera 34, the rearcentral (C) camera 31, the rear central right (CR) camera 35, and therear right (R) camera 33. These five cameras are installed from the leftside to the right side of the rear of the automobile 10.

As illustrated in FIG. 16, the five cameras image the rear of theautomobile from different viewpoints. Parts of the imaging regionoverlap with each other. Smoother image switching display is madepossible by sequentially switching and displaying images captured by thefive cameras having these different imaging viewpoints in accordancewith the behavior of a driver, for example, movements of the head.

FIG. 17 illustrates an example of switching images in a case where morecameras (c1 to cn=15) are disposed in the rear of the automobile 10 andimages are switched.

A driver can observe images smoothly changing like a moving image bysequentially switching manual images captured by a plurality of camerasin this way.

FIG. 18 illustrates an example in which the number of cameras installedin the rear of the automobile 10 is set smaller than that in the examplein FIG. 17. Although images may be switched with a small number ofcameras in this way, parallax difference between the cameras causesintermittent images. In such a case, processing of calibrating eachcamera captured image is performed to set and display an infinite point101 of each image at a fixed position of the image. The processingenables inhibition of subject blurring generated at the time ofswitching images, and enables an image viewer (driver) to observe asmoothly changing image without unnatural change.

Furthermore, image correction of sweeping and shifting an image in alateral direction at a neighboring point in an image in accordance withmovements of the head of the driver may be performed. The processingenables the driver to visually and intuitively recognize the switchingbetween the camera images.

Furthermore, in a case where a small number of cameras are installed inthe rear of the automobile 10, and the parallax difference between thecameras causes intermittent images, a virtual viewpoint image of amiddle viewpoint between two adjacent cameras may be synthesized anddisplayed on the basis of images captured by the two adjacent cameras.

That is, image display control of switching an image output to thedisplay unit to any of images captured by a plurality of cameras or avirtual viewpoint composite image is performed.

FIG. 19 illustrates an example of generating a virtual viewpoint image.Here, (1) camera C3 captured image and (4) camera C4 captured image inFIG. 19 are images actually captured by adjacent cameras C3 and C4.

Virtual viewpoint composite images of (2) and (3) are generated on thebasis of these two actually captured images.

Here, (2) virtual viewpoint composite image is a composite imagegenerated in the proportion of (1) camera C3 captured image of 65% and(4) camera C4 captured image of 35%.

Here, (3) virtual viewpoint composite image is a composite imagegenerated in the proportion of (1) camera C3 captured image of 35% and(4) camera C4 captured image of 65%.

Smoothly changing images can be presented for a driver by sequentiallyswitching and displaying the four images (1) to (4) in FIG. 19.

What is important for the driver is, however, not to visually check thedetails of an image at a halfway viewpoint between cameras but to judgea rear situation. The driver sometimes cannot judge a switched situationby just instantly viewing an acquired image, at a rear C cameraposition, captured by the rear C camera 31 at the center of the rear ofthe car and an image taken from a camera disposed on a side surface.Switching and presenting a plurality of halfway images for the driverenables the driver to sensorily and naturally recognize image switching.Thus, confusion in thinking at the time when instant situation judgmentis necessary can be avoided.

Switching based on, for example, the orientation of a face and directionof line-of-sight, switching in accordance with a so-called gesture suchas hand or neck shaking, and the like can be used for switching images.Switching with a gesture can give an advantage that it is unnecessary tomaintain an improper posture. Note, however, that, in a case where animage is switched without depending on the orientation of a face or thedirection of a line-of-sight, the observation posture of the driver isthe same. Thus, information indicating switching and transition of animage, for example, auxiliary information such as a switching mark ispreferably displayed.

In the processing of the present disclosure, a means for properlychecking the rear without occlusion is provided by providing cameraimages of the rear of a vehicle obtained from a plurality of differentviewpoints. The camera images having the different viewpoints are madeto have continuity. An image on the halfway of switching is simulativelygenerated by using human visual characteristics, and presented on amonitor. In this way, the driver can sensorily grasp viewpoint movementsfrom the image. In formation can be provided while reduction or loss ingrasping a situation is inhibited. That is, ergonomically, viewpointscan be grasped well by the driver viewing an image that warps betweenchanging cameras having different viewpoints.

FIG. 20 illustrates an example in which a camera position icon 102 issuperimposed and displayed on an image displayed on the display unit 20.The camera position icon 102 indicates a camera imaging viewpoint. Inthis way, displaying the camera position icon 102, which indicates theposition of a camera that has captured an image, together with the imageenables the driver to immediately check an image captured position, andmore quickly recognize the position of a subject in the image.

FIG. 21 illustrates an example in which a rear left end camera image anda left side camera image are displayed side by side on the display unit20. The rear left end camera image is an image captured by a rear leftend camera installed at a left end of the rear of the automobile 10. Theleft side camera image is an image captured by the left side camera 12.Such display enables the situation on the left side of the automobile 10to be certainly grasped.

Note that, similarly, displaying an image captured by a rear right endcamera installed at a right end of the rear of the automobile 10 and animage captured by the right side camera 13 side by side enables thesituation on the right side of the automobile 10 to be certainlygrasped.

Although FIG. 21 illustrates an example in which a pictogramrepresenting a camera is displayed in the vicinity of a vehicle bumper,a simple color marker, the entire screen, or right and left frames maybe displayed thick, and made blinking. Flow zebra caution may be simplydisplayed. Another caution calling display may be made as long as thedriver can intuitively understand camera arrangement at the time oflooking at the screen. Furthermore, a plurality of displays may becombined. Furthermore, configuration, in which switching to a displaymethod that does not impair intuitive understanding can be made, can beadopted in accordance with the preference of the driver.

Note that, for example, as illustrated in FIG. 22, the following twoinstallation aspects are mainly adopted as an aspect of installing aplurality of cameras disposed in the rear of the automobile 10.

(a) Radial Arrangement

(b) Linear Arrangement

Here, (a) radial arrangement is an example in which cameras are disposedon a curve having a predetermined curvature, and is characterized bybeing able to capture an image in a wider range. In contrast, (b) lineararrangement has an advantage that, although the imaging range isnarrowed, an amount of deformation of an object in each image is small,and an amount of correction processing for a display image is reduced.Note that, although, contrary to the radial type of (a), arrangement ina direction of convergence to a rear neighboring point is possible in aconvergence arrangement, the installation aspect has a disadvantage thatcontinuity in directions to the left side camera 12 and the right sidecamera 13 cannot be obtained. An image of actually installed camera isused as a projection image of a central projection image, and it is notnecessary to dispose an optical axis in each direction. An image may beconverted from a fisheye projection image, and image conversion may beused so that an optical axis of a virtual central projection imagecorresponds to an optical axis in accordance with each installationaspect.

4. Image Processing Apparatus and Sequence of Processing Executed byMoving Apparatus

Next, an image processing apparatus and a sequence of processingexecuted by a moving apparatus of the present disclosure will bedescribed with reference to the flowchart of FIG. 23.

The processing of the flow in FIG. 23 and subsequent figures is executedin a moving apparatus or an image processing apparatus mounted in themoving apparatus. For example, the processing is performed under thecontrol of a data processing unit including a processor such as a CPU.The processor has a function of executing a program in accordance withthe program stored in a storage unit in an apparatus.

The processing of each step of the flow in FIG. 23 will be describedbelow.

(Step S101)

First, in Step S101, the data processing unit detects a face,line-of-sight direction, or a gesture of a driver, or informationregarding operations of the driver.

This is processing of, for example, the data processing unit 60 in FIG.12 inputting detection information from the driver information detectionunit 52 and information regarding operations of the driver to theoperation unit.

(Step S102)

Next, in Step S102, the data processing unit determines whether or notthe input of a face, change of a line-of-sight direction, or a gestureof a driver, or information regarding operations of the driver isdetected.

If the input is detected, the processing proceeds to Step S103. If theinput is not detected, the detection processing of Step S101 iscontinued.

(Step S103)

Next, in Step S103, the data processing unit selects a camera capturedimage in a direction in accordance with the face, the line-of-sightdirection, or the gesture of the driver, or the operation information,which has been input in Step S102.

Note that, for example, if there is no image that completely matches aline-of-sight direction of the driver, an image in a closest directionis selected.

For example, if the face of, the line-of-sight, or the gesture thedriver, or the input to the operation unit is in a left direction, animage captured by a camera (rear L camera) on the left side of the rearof the automobile 10 is selected.

Furthermore, for example, if the face, the line-of-sight, or the gestureof the driver, or the input to the operation unit is in a centraldirection, an image captured by a camera (rear C camera) in the centerof the rear of the automobile 10 is selected. Alternatively, forexample, if the face, the line-of-sight, or the gesture of the driver,or the input to the operation unit is in a right direction, an imagecaptured by a camera (rear R camera) on the right side of the rear ofthe automobile 10 is selected.

(Step S104)

Next, in Step S104, the data processing unit displays the image that hasbeen selected in Step S103 on the display unit.

Note that various settings can be adopted for an image displayed on thedisplay unit. The various settings include, for example, a setting inwhich only a selected rear camera captured image is displayed and asetting in which two side camera images captured by right and left sidecameras are displayed together.

In the flow in FIG. 23, processing of selecting an image in the nearestdirection is performed in a case where, for example, there is no imagethat completely matches a line-of-sight direction of the driver at thetime when a camera captured image in a direction in accordance with theface, the line-of-sight direction, or the gesture of the driver, or theoperation information, which has been input in Step S102, is selected.

Next, a sequence of processing of synthesizing images in theline-of-sight direction of the driver in such a case, that is, in a casewhere there is no image that completely matches the line-of-sightdirection of the driver will be described with reference to theflowchart of FIG. 24.

The processing of each step of the flow in FIG. 24 will be describedbelow.

(Step S201)

First, in Step S201, the data processing unit detects a face, aline-of-sight direction, or a gesture of a driver, or informationregarding operations of the driver.

This is processing of, for example, the data processing unit 60 in FIG.12 inputting detection information from the driver information detectionunit 52 and information regarding operations of the driver to theoperation unit.

(Step S202)

Next, in Step S202, the data processing unit determines whether or notthe input of a face, change of a line-of-sight direction, a gesture of adriver, or information regarding operations of the driver is detected.

If the input is detected, the processing proceeds to Step S203. If theinput is not detected, the detection processing of Step S201 iscontinued.

(Step S203)

Next, in Step S203, the data processing unit determines the presence orabsence of a camera captured image in a direction in accordance with theface, change of the line-of-sight direction, or the gesture of thedriver, or the information regarding operations of the driver, which hasbeen input in Step S202.

If there is a captured image in the matching direction, the processingproceeds to Step S204.

In contrast, if there is no captured image in the matching direction,the processing proceeds to Step S206.

(Step S204)

In a case where it is determined in Step S203 that there is the cameracaptured image in a direction in accordance with the face, change of theline-of-sight direction, or the gesture of the driver, or theinformation regarding operations of the driver, which has been input inStep S202, the processing of Steps S204 and S205 is executed. Thedisplay is mainly suitable for a case where cameras are adjacentlydisposed at the time of switching and an image is not described in abound. If imaging directions of switching images are far away from eachother, switching in accordance with the later-described Step S206 isperformed.

In Step S204, a camera captured image in a direction in accordance withthe face, the line-of-sight direction, or the gesture of the driver, orthe operation information, which has been input in Step S202, isselected.

For example, if the face, the line-of-sight, or the gesture of thedriver, or the input to the operation unit is in a left direction, animage captured by a camera (rear L camera) on the left side of the rearof the automobile 10 is selected.

Furthermore, for example, if the face, the line-of-sight, or the gestureof the driver, or the input to the operation unit is in a centraldirection, an image captured by a camera (rear C camera) in the centerof the rear of the automobile 10 is selected. Alternatively, forexample, if the face, the line-of-sight, or the gesture of the driver,or the input to the operation unit is in a right direction, an imagecaptured by a camera (rear R camera) on the right side of the rear ofthe automobile 10 is selected.

(Step S205)

Next, in Step S205, the data processing unit displays the image that hasbeen selected in Step S204 on the display unit.

Note that various settings can be adopted for an image displayed on thedisplay unit. The various settings include, for example, a setting inwhich only a selected rear camera captured image is displayed and asetting in which two side camera images captured by right and left sidecameras are displayed together.

(Step S206)

In contrast, in a case where it is determined in Step S203 that there isno camera captured image in a direction in accordance with the face,change of the line-of-sight direction, or the gesture of the driver, orthe information regarding operations of the driver, which has been inputin Step S202, the processing of Steps S206 and S207 is executed.

First, in Step S206, processing of generating, as a composite image, acamera captured image in a direction in accordance with the face of,change of the line-of-sight direction, or the gesture of the driver, orthe information regarding operations of the driver, which has been inputin Step S202 is executed.

The composite image generating processing corresponds to, for example,the image synthesizing processing described above with reference to FIG.19.

The image synthesizing processing can be performed by, for example,simply adding a weight to images actually captured by adjacent cameras.Furthermore, a middle virtual image may be generated by detecting amoving object for each region and performing correction on the basis ofthe detection information. Note that information loss of a driver isprevented by presenting a composite image of a middle viewpoint betweenactually captured images to the driver. Specifically, for example, ifthe captured image is switched from the rear central (C) camera 31 to aside camera, the image greatly changes in an instant. The compositeimage is effective in preventing the information loss of the driver,which may occur in such a case.

Note that, for example, input of the face, the line-of-sight direction,or gesture of the driver, or image selection indication in Step S206makes various movements such as moderate movements and sharp movements.Steps 206 and 207 are executed as steps of generating one or two or moreplurality of middle images, that is, composite images in accordance withthe detected movement.

(Step S207)

Next, in Step S207, the data processing unit displays the compositeimage that has been generated in Step S206 on the display unit. Inaddition, the driver can grasp the camera position of the display imageby superimposing and displaying the description camera position on ascreen of the display unit 20 in FIG. 21 at the time when the compositeimage in Step S207 is output to the display unit. Note that varioussettings can be adopted for an image displayed on the display unit. Thevarious settings include, for example, a setting in which only aselected rear camera captured image is displayed and a setting in whichtwo side camera images captured by right and left side cameras aredisplayed together.

Furthermore, in the description of the sequence, images are switched by,for example, movement of face, line-of-sight, and head based on the willof the driver, a gesture, or an operation. In contrast, as describedabove with reference to FIGS. 7, 8, and 9, an object behind a vehicle inthe rear, that is, an object in an occlusion region may be detected, anda warning prompting a driver to visually check details may be displayedas a trigger for the driver to check a side camera image at the timewhen the driver looks at the rear central (C) camera 31. The warningincludes a zebra pattern and a blinking superposition warning display atthe rear object boundary that generates the occlusion.

That is, if, for example, a large vehicle approaches behind, and only animage captured by a central camera is used, an approaching vehicle orthe like behind the large vehicle is easily overlooked due to theocclusion. In such a case, a warning indicating that an object is in theocclusion region is displayed. The warning prompts the driver to performchecking with images captured by cameras having different viewpoints.The processing enables the driver to change the camera viewpoints andcertainly check the rear situation even in a case where there isocclusion, thereby improving safety.

Note that an image may be switched in configuration in which differentcamera captured images in a single bound are provided. Preferably, animage having a virtual viewpoint between cameras is output betweendifferent camera captured images, and display with image transition andcontinuity is performed. The processing prevents loss in grasping asituation of the driver, and achieves safe checking of the rear side.

5. Configuration Example of Image Processing Apparatus

Next, a hardware configuration example of an image processing apparatusthat executes the above-described processing will be described.

FIG. 25 illustrates a hardware configuration example of the imageprocessing apparatus.

A central processing unit (CPU) 501 functions as a data processing unitthat executes various pieces of processing in accordance with a programstored in a read only memory (ROM) 502 or a storage unit 508. Forexample, processing is performed in accordance with the sequencedescribed in the above-described embodiment.

For example, a program to be executed by the CPU 501 and data are storedin a random access memory (RAM) 503. These CPU 501, ROM 502, and RAM 503are mutually connected by a bus 504.

The CPU 501 is connected to an input/output interface 505 via the bus504. An input unit 506 and an output unit 507 are connected to theinput/output interface 505. The input unit 506 includes, for example, asituation data acquisition unit such as various switches, a keyboard, atouch panel, a mouse, a microphone, a sensor, a camera, and a GPS. Theoutput unit 507 includes, for example, a display and a speaker.

The CPU 501 inputs, for example, a command and situation data input fromthe input unit 506, executes various pieces of processing, and outputsthe processing result to, for example, the output unit 507.

The storage unit 508 connected to the input/output interface 505includes, for example, a hard disk, and stores a program to be executedby the CPU 501 and various pieces of data. A communication unit 509functions as a transmission/reception unit for data communication via anetwork such as the Internet or a local area network, and communicateswith an external apparatus.

A drive 510 connected to the input/output interface 505 drives aremovable medium 511 including a magnetic disk, an optical disk, amagneto-optical disk, and a semiconductor memory such as a memory card.The drive 510 records or reads data.

6. Summary of Configuration of Present Disclosure

The embodiments of the present disclosure have been described in detailabove with reference to the specific embodiments. It is, however,obvious that those skilled in the art can modify or substitute theembodiments without departing from the gist of the present disclosure.That is, the present invention has been disclosed in the form ofexemplification, and should not be limitedly interpreted. The claimsshould be considered in order to determine the gist of the presentdisclosure.

Note that the technology disclosed in the specification can haveconfigurations as follows.

(1) An image processing apparatus including a data processing unit thatexecutes image display control of:

inputting

driver information indicating behavior of a driver of a moving apparatusand

images captured by a plurality of cameras that images a situation aroundthe moving apparatus from different viewpoints; and

switching an image output to a display unit to any of the imagescaptured by the plurality of cameras or a virtual viewpoint compositeimage in accordance with the driver information.

(2) The image processing apparatus according to (1),

in which the plurality of cameras includes a plurality of rear camerasinstalled in a rear of the moving apparatus, and

the data processing unit performs switching to and display of any ofimages captured by the plurality of rear cameras or a virtual viewpointcomposite image in accordance with the driver information.

(3) The image processing apparatus according to (1),

in which the plurality of cameras includes: a plurality of rear camerasinstalled in a rear of the moving apparatus; and a side camera installedon a side surface of the moving apparatus, and

the data processing unit performs switching to and display of any ofimages captured by the plurality of rear cameras and the side camera ora virtual viewpoint composite image in accordance with the driverinformation.

(4) The image processing apparatus according to any one of (1) to (3),

in which the data processing unit displays one rear camera capturedimage and an image captured by a side camera on the display unit side byside, the rear camera captured image having been selected from imagescaptured by a plurality of rear cameras installed in a rear of themoving apparatus in accordance with the driver information, the sidecamera being installed on a side surface of the moving apparatus.

(5) The image processing apparatus according to any one of (1) to (4),

in which the driver information includes information obtained bydetecting a direction of a face or line-of-sight of the driver, and

the data processing unit displays an image in a direction correspondingto the direction of a face or line-of-sight of the driver on the displayunit.

(6) The image processing apparatus according to any one of (1) to (4),

in which the driver information includes information obtained bydetecting a gesture of the driver, and

the data processing unit displays an image in a direction in accordancewith a gesture of the driver on the display unit.

(7) The image processing apparatus according to any one of (1) to (4),

in which the data processing unit displays an image in a direction inaccordance with input of the driver to an operation unit on the displayunit.

(8) The image processing apparatus according to any one of (1) to (7),

in which the virtual viewpoint composite image is a composite imagebased on images captured by a plurality of cameras, and is a compositeimage corresponding to a captured image from a virtual viewpoint betweenimaging viewpoints of the plurality of cameras.

(9) The image processing apparatus according to any one of (1) to (8),

in which the driver information includes information obtained bydetecting a direction of a face or line-of-sight of the driver, and

the data processing unit generates an image in a direction correspondingto the direction of a face or line-of-sight of the driver bysynthesizing images captured by a plurality of cameras, and displays theimage on the display unit.

(10) The image processing apparatus according to any one of (1) to (9),

in which the data processing unit executes image calibration processingof matching positions of infinite points of images at a time ofswitching display of an image on the display unit.

(11) A moving apparatus including:

a plurality of cameras that images a situation around the movingapparatus from different viewpoints;

a driver information detection unit that detects driver informationindicating behavior of a driver of the moving apparatus; and

a data processing unit that inputs the driver information and imagescaptured by the plurality of cameras, and switches an image output to adisplay unit to any of the images captured by the plurality of camerasor a virtual viewpoint composite image in accordance with the driverinformation.

(12) The moving apparatus according to (11),

in which the plurality of cameras includes a plurality of rear camerasinstalled in a rear of the moving apparatus, and

the data processing unit performs switching to and display of any ofimages captured by the plurality of rear cameras or a virtual viewpointcomposite image in accordance with the driver information.

(13) The moving apparatus according to (11) or (12),

in which the driver information detection unit detects a direction of aface or line-of-sight of the driver, and

the data processing unit displays an image in a direction correspondingto the direction of a face or line-of-sight of the driver on the displayunit.

(14) An image processing method executed in an image processingapparatus,

in which a data processing unit executes image display control of:

inputting

driver information indicating behavior of a driver of a moving apparatusand

images captured by a plurality of cameras that images a situation aroundthe moving apparatus from different viewpoints; and

switching an image output to a display unit to any of the imagescaptured by the plurality of cameras or a virtual viewpoint compositeimage in accordance with the driver information.

(15) A display image control method executed in a moving apparatus,comprising:

an image capturing step in which a plurality of cameras images asituation around the moving apparatus from different viewpoints;

a driver information detection step in which a driver informationdetection unit detects driver information indicating behavior of adriver of the moving apparatus; and

an image display control step in which a data processing unit inputs thedriver information and images captured by the plurality of cameras, andswitches an image output to a display unit to any of the images capturedby the plurality of cameras or a virtual viewpoint composite image inaccordance with the driver information.

(16) A program for executing image processing in an image processingapparatus, causing a data processing unit to execute image displaycontrol of:

inputting

driver information indicating behavior of a driver of a moving apparatusand

images captured by a plurality of cameras that images a situation aroundthe moving apparatus from different viewpoints; and

switching an image output to a display unit to any of the imagescaptured by the plurality of cameras or a virtual viewpoint compositeimage in accordance with the driver information.

Furthermore, a series of processing described in the specification canbe executed by hardware, software, or a combined configuration of both.In a case where the processing is executed by software, a program, inwhich a processing sequence is recorded, can be executed by beinginstalled in a memory in a computer incorporated in dedicated hardwareor being installed in a general-purpose computer capable of executingvarious pieces of processing. For example, a program can bepreliminarily recorded in a recording medium. In addition to beinginstalled in a computer from a recording medium, a program can bereceived via a network such as a local area network (LAN) and theInternet, and installed in a recording medium such as a built-in harddisk.

Note that the various pieces of processing described in thespecification may be not only executed in chronological order inaccordance with the description, but executed in parallel orindividually in accordance with the processing capability or needs of anapparatus that executes processing. Furthermore, a system in thespecification has configuration of a logical set of a plurality ofapparatuses. The system is not limited to a system in which apparatuseshaving each configuration are placed in the same housing.

INDUSTRIAL APPLICABILITY

As described above, according to the configuration of one embodiment ofthe present disclosure, configuration, in which images output to adisplay unit are switched and displayed in accordance with the behaviorof a driver, such as movements of the head of the driver, is achieved.

Specifically, for example, driver information indicating the behavior ofa driver of a moving apparatus and images captured by a plurality ofcameras that images the situation around the moving apparatus fromdifferent viewpoints are input. Images output to the display unit areswitched in accordance with driver information. The plurality of camerasis, for example, a plurality of rear cameras installed in the rear ofthe moving apparatus. For example, a direction of the face orline-of-sight of the driver is detected. An image in a directioncorresponding to the detected face or direction of line-of-sight of thedriver is selected as an output image, and displayed on a display unit.Alternatively, an image in a direction indicated by a gesture of thedriver is selected, and displayed on the display unit. Furthermore, acomposite image having a middle viewpoint is appropriately generated anddisplayed at the time when different camera captured images areswitched, whereby loss of the driver in grasping a situation isprevented.

The configuration achieves configuration in which images output to thedisplay unit are switched and displayed in accordance with the behaviorof the driver, such as movements of the head of the driver.

REFERENCE SIGNS LIST

-   10 Automobile-   11 Rear camera-   12 Left side camera-   13 Right side camera-   20 Display unit-   21 Rear camera image-   22 Left side camera image-   23 Right side camera image-   25 Composite image-   31 Rear central (C) camera-   32 Rear left (L) camera-   33 Rear right (R) camera-   34 Rear central left (CL) camera-   35 Rear central right (CR) camera-   41 Rear-camera-image switching display region-   50 Driver-   52 Driver information detection unit-   60 Data processing unit-   101 Infinite point-   102 Camera position icon-   501 CPU-   502 ROM-   503 RAM-   504 Bus-   505 Input/output interface-   506 Input unit-   507 Output unit-   508 Storage unit-   509 Communication unit-   510 Drive-   511 Removable medium

1. An image processing apparatus comprising a data processing unit thatexecutes image display control of: inputting driver informationindicating behavior of a driver of a moving apparatus and imagescaptured by a plurality of cameras that images a situation around themoving apparatus from different viewpoints; and switching an imageoutput to a display unit to any of the images captured by the pluralityof cameras or a virtual viewpoint composite image in accordance with thedriver information.
 2. The image processing apparatus according to claim1, wherein the plurality of cameras includes a plurality of rear camerasinstalled in a rear of the moving apparatus, and the data processingunit performs switching to and display of any of images captured by theplurality of rear cameras or a virtual viewpoint composite image inaccordance with the driver information.
 3. The image processingapparatus according to claim 1, wherein the plurality of camerasincludes: a plurality of rear cameras installed in a rear of the movingapparatus; and a side camera installed on a side surface of the movingapparatus, and the data processing unit performs switching to anddisplay of any of images captured by the plurality of rear cameras andthe side camera or a virtual viewpoint composite image in accordancewith the driver information.
 4. The image processing apparatus accordingto claim 1, wherein the data processing unit displays one rear cameracaptured image and an image captured by a side camera on the displayunit side by side, the rear camera captured image having been selectedfrom images captured by a plurality of rear cameras installed in a rearof the moving apparatus in accordance with the driver information, theside camera being installed on a side surface of the moving apparatus.5. The image processing apparatus according to claim 1, wherein thedriver information includes information obtained by detecting adirection of a face or line-of-sight of the driver, and the dataprocessing unit displays an image in a direction corresponding to thedirection of a face or line-of-sight of the driver on the display unit.6. The image processing apparatus according to claim 1, wherein thedriver information includes information obtained by detecting a gestureof the driver, and the data processing unit displays an image in adirection in accordance with a gesture of the driver on the displayunit.
 7. The image processing apparatus according to claim 1, whereinthe data processing unit displays an image in a direction in accordancewith input of the driver to an operation unit on the display unit. 8.The image processing apparatus according to claim 1, wherein the virtualviewpoint composite image is a composite image based on images capturedby a plurality of cameras, and is a composite image corresponding to acaptured image from a virtual viewpoint between imaging viewpoints ofthe plurality of cameras.
 9. The image processing apparatus according toclaim 1, wherein the driver information includes information obtained bydetecting a direction of a face or line-of-sight of the driver, and thedata processing unit generates an image in a direction corresponding tothe direction of a face or line-of-sight of the driver by synthesizingimages captured by a plurality of cameras, and displays the image on thedisplay unit.
 10. The image processing apparatus according to claim 1,wherein the data processing unit executes image calibration processingof matching positions of infinite points of images at a time ofswitching display of an image on the display unit.
 11. A movingapparatus comprising: a plurality of cameras that images a situationaround the moving apparatus from different viewpoints; a driverinformation detection unit that detects driver information indicatingbehavior of a driver of the moving apparatus; and a data processing unitthat inputs the driver information and images captured by the pluralityof cameras, and switches an image output to a display unit to any of theimages captured by the plurality of cameras or a virtual viewpointcomposite image in accordance with the driver information.
 12. Themoving apparatus according to claim 11, wherein the plurality of camerasincludes a plurality of rear cameras installed in a rear of the movingapparatus, and the data processing unit performs switching to anddisplay of any of images captured by the plurality of rear cameras or avirtual viewpoint composite image in accordance with the driverinformation.
 13. The moving apparatus according to claim 11, wherein thedriver information detection unit detects a direction of a face orline-of-sight of the driver, and the data processing unit displays animage in a direction corresponding to the direction of a face orline-of-sight of the driver on the display unit.
 14. An image processingmethod executed in an image processing apparatus, wherein a dataprocessing unit executes image display control of: inputting driverinformation indicating behavior of a driver of a moving apparatus andimages captured by a plurality of cameras that images a situation aroundthe moving apparatus from different viewpoints; and switching an imageoutput to a display unit to any of the images captured by the pluralityof cameras or a virtual viewpoint composite image in accordance with thedriver information.
 15. A display image control method executed in amoving apparatus, comprising: an image capturing step in which aplurality of cameras images a situation around the moving apparatus fromdifferent viewpoints; a driver information detection step in which adriver information detection unit detects driver information indicatingbehavior of a driver of the moving apparatus; and an image displaycontrol step in which a data processing unit inputs the driverinformation and images captured by the plurality of cameras, andswitches an image output to a display unit to any of the images capturedby the plurality of cameras or a virtual viewpoint composite image inaccordance with the driver information.
 16. A program for executingimage processing in an image processing apparatus, causing a dataprocessing unit to execute image display control of: inputting driverinformation indicating behavior of a driver of a moving apparatus andimages captured by a plurality of cameras that images a situation aroundthe moving apparatus from different viewpoints; and switching an imageoutput to a display unit to any of the images captured by the pluralityof cameras or a virtual viewpoint composite image in accordance with thedriver information.